A New Generic Taxonomy of Malware Behavioural Detection and Removal Techniques

Modern malware has become a major threat to today’s Internet communications. The threat can infiltrate hosts using a variety of methods, such as attacks against known software vulnerabilities, hidden functionality in regular programs, drive-by download from unsafe web sites, and so forth. Matching a file stream against a known virus pattern is a fundamental technique for detecting viruses. With the popularity and variety of malware attack over the Internet, computer virus protection companies need to constantly update new virus signatures in their virus definition databases. However, the increasing size of the signature database can only detect known virus but cannot defend against new variants of malware. In this paper, we present an overview of the detection of modern malware focuses on suspect behavioural patterns. Contrary to classical heuristic engines which focus on the detection of encrypted malware samples, we integrate a known packer detector as well as unpacking routines to circumvent the protection techniques used by most of the modern malware. We believe that many obfuscated techniques used by malware authors are available on the Internet. More precisely, the use of known packer removals would strip out the packer protection with our dedicated decryption routines. Our apprehensive program is based on the integration of both static heuristic and emulator approaches; however, they do not necessarily have to serve as a complement for each other. Static heuristic scanner involves static extraction, which is relying on byte signature to identify a dedicated viral signature. Emulator can execute the arbitrary code from the instance and would trace the instance’s body code in a virtual environment. It can be used to combat any protection code, regardless of the complexity of the protection algorithm. Fragments of virus body could be detected while the execution is in a decrypted virus body. Lastly, we present experimental results that indicate our proposed technique can provide good performance against obfuscated malware. Through this study, we hope to help security researchers understand our defence approach and give some directions for future research.